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Abstract 


The  current  s-curve  method  of  cost-risk  analysis  for  major  DoN  acquisitions 
projects  does  not  accurately  estimate  actual  cost  when  the  program  reaches  full-rate 
production.  Another,  sometimes  more  effective  method  of  measuring  cost  risk,  is 
the  enhanced  scenario-based  method  (eSBM)  of  risk  analysis.  The  reason  that  cost 
estimations  based  on  Milestone  B  costs  are  inaccurate  is  that  little,  if  any,  real 
information  about  the  project  is  known  at  that  point  in  the  project.  eSBM  allows 
managers  a  less  statistically  tasking  method  of  determining  cost  risk  for  a  project, 
while  still  maintaining  the  requirements  of  the  Weapons  System  Acquisitions  Reform 
Act  (WSARA)  of  2009.  In  measuring  the  usefulness  of  eSBM,  the  key  factors  should 
focus  on  the  acquisition  strategy  being  used  for  the  project  and  the  time  frame  from 
Milestone  B  to  later  milestones.  I  hypothesized  that  different  acquisition  strategies 
will  yield  different  levels  of  success  in  estimating  cost  risk  for  eSBM. 
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I. 


INTRODUCTION 


A.  BACKGROUND 

The  current  method  of  choice  for  developing  risk-adjusted  cost  estimates  is 
the  Monte  Carlo  simulation  method,  which  sums  probability-based  cost  distributions 
at  the  work  breakdown  structure  (WBS)  element  level  to  produce  a  cumulative 
distribution  for  overall  program  cost.  The  confidence  level  from  the  cumulative 
distribution  is  the  resulting  “risk”  for  the  estimated  cost.  This  Monte  Carlo  method  is 
also  referred  to  within  the  cost-estimating  community  as  the  s-curve  method  of  cost- 
risk  analysis,  and,  although  it  is  the  current  method  of  choice  for  developing  risk- 
adjusted  cost  estimates,  it  leaves  several  of  the  following  problems  unmanaged: 

1 .  Programs  do  not  behave  as  the  cost-estimation  model  had  forecast. 
The  cost  risk  becomes  a  second-order  guess  based  on  another  guess. 

2.  This  model  does  not  bound  possible  risk  areas  on  either  high  or  low 
likelihood  probability.  By  using  this  method,  the  left  and  right  tails  of  the 
cumulative  distribution  are  weighted  equally  with  the  mean. 

3.  The  model  assumes  that  the  consequences  for  all  unfavorable 
outcomes  equally  affect  the  cost  model. 

4.  The  model  requires  in-depth  knowledge  of  statistical  analysis  and 
processes. 

The  s-curve  method  of  cost-risk  analysis  for  major  Department  of  Defense 
(DoD)  and  Department  of  the  Navy  (DoN)  acquisitions  projects,  so-called  major 
defense  acquisition  programs  (MDAPs),  does  not  accurately  estimate  actual  cost 
from  the  pre-Milestone  B  stages  of  the  acquisitions  process  to  the  phase  at  which 
the  program  reaches  full-rate  production;  thus,  the  s-curve  method  produces 
unreliable  cost-risk  analysis. 

One  of  the  reasons  that  cost  estimates  at  Milestone  B  are  inaccurate  is  that 
very  little,  if  any,  real  information  about  the  project  is  known  other  than  the 
conceptual  capabilities  required.  The  enhanced  scenario-based  method  (eSBM) 
makes  risk  assessments  without  the  need  for  details  about  the  program,  while  still 
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being  grounded  in  historical  facts  and  data.  eSBM  allows  managers  a  less 
statistically  taxing  method  of  determining  cost  risk  for  a  project. 

B.  PURPOSE 

The  key  factors  in  measuring  the  usefulness  of  eSBM  should  be  to  focus  on 
the  acquisition  strategy  being  used  for  the  project  and  the  time  frame  from  Milestone 
B  to  later  milestones.  In  this  thesis,  I  hypothesized  that  different  acquisition 
strategies  yield  different  levels  of  variation  in  estimating  cost  risk  for  eSBM. 
Furthermore,  I  hypothesized  that  the  longer  a  program  takes  to  reach  full-rate 
production  from  Milestone  B,  the  more  cost  risk  there  is  for  that  program. 

The  eSBM  method  is  not  meant  to  replace  the  use  of  statistical  analysis  to 
determine  cost  risk;  it  is  intended  to  provide  an  alternative  to  the  current  standard. 

By  having  an  additional  method  of  risk  analysis,  cost  estimators  and  program 
managers  are  provided  with  yet  another  risk  profile  with  which  to  compare  and 
measure  risk. 

C.  THE  QUESTION 

No  definitive  research  links  the  effects  of  acquisition  strategy  and/or  program 
length  to  program  cost  risk  and  the  Monte  Carlo  method  of  cost-risk  analysis  makes 
no  attempt  to  factor  in  acquisition  strategy  or  the  time  horizon  for  the  program.  On 
the  other  hand,  the  eSBM  makes  an  effort  to  reduce  statistical  uncertainty  in 
estimating  costs  and  reducing  the  risk  of  cost  overrun.  However,  it  falls  short  in  the 
areas  of  acquisition  strategy  and  time  horizon.  I  hypothesized  that  different 
acquisition  strategies  yield  different  levels  of  risk  in  estimating  cost  risk  for  eSBM. 


D.  WHO  BENEFITS 

More  accurate  risk  analysis  benefits  every  level  of  DoD  decision-making.  The 
following  examples  outline  some  of  these  benefits. 
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■  By  providing  an  accurate  assessment  of  a  program’s  cost  risk,  the 
senior  acquisition  managers  and  Milestone  Decision  Authorities 
(MDAs)  will  be  able  to  manage  program  risk  early  on  and  avoid 
potentially  overly  risky  programs.  Having  another  cost-risk  profile  will 
allow  for  more  rational  debates  during  the  analysis  of  alternatives 
(AOA)  stage  of  the  acquisition  process.  By  weeding  out  programs  with 
a  high  likelihood  of  failure,  more  funds  will  be  available  for  other 
programs  that  are  likely  to  succeed. 

■  Program  executive  offices  (PEOs)  and  program  managers  (PMs)  will 
be  able  to  focus  attention  on  areas  of  highest  risk  sooner  in  the 
program,  thus  detecting  the  potential  effects  on  cost,  schedule,  and 
performance  before  problems  arise.  The  more  accurate  cost-risk 
assessment  also  allows  for  realistic  decisions  about  how  to  manage 
the  program  as  a  whole. 

■  Cost  estimators  would  be  provided  with  a  method  of  analysis  that  is 
founded  in  historical  data  without  the  rigorous  statistical  analysis.  Due 
to  the  complex  statistical  analysis  methods  needed  to  produce 
accurate  Monte  Carlo  simulations,  cost  estimations  are  subject  to 
additional  scrutiny  from  statisticians.  The  eSBM  method  removes  this 
statistical  scrutiny  while  still  maintaining  the  desired  outcome  in  terms 
familiar  to  decision-makers. 

In  a  dynamic  world  of  fiscal  constraints,  accurate  cost-estimating  risk 
assessments  are  even  more  crucial. 

E.  THE  NEXT  STEPS 

In  the  next  chapter,  I  describe  how  the  eSBM  was  developed,  how  its 
parameters  were  derived,  and  how  it  is  used  for  cost-risk  analysis.  In  that  section,  I 
include  a  description  of  the  methods  used  to  construct  the  coefficients  of  variation 
(CV)  for  the  different  cost  drivers.  Additionally,  I  explain  how  to  translate  the  eSBM 
results  into  terms  familiar  to  the  acquisition  community. 

In  this  thesis,  I  present  my  research  on  how  acquisition  strategy  and  program 
length  affect  the  cost-risk  profile  of  MDAPs.  In  this  research,  I  derived  the  cost 
growth  factors  (CGF)  and  coefficients  of  variation  by  using  historical  data.  After  the 
CVs  were  formulated,  I  tested  them  for  validity  and  translated  them  into  a  cost-risk 
profile. 
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LITERATURE  REVIEW 


A.  INTRODUCTION 

In  2005,  Paul  Garvey  introduced  the  scenario-based  method  (SBM)  as  an 
alternative  to  the  s-curve  method  of  risk-adjusted  cost  estimation;  he  described  it  as 
an  effective,  less  statistically  intensive  method  of  determining  cost-risk  analysis 
(Garvey,  2005).  The  Air  Force  introduced  the  SBM  in  the  2007  Cost  Risk  and 
Uncertainty  Handbook  as  an  acceptable  method  to  calculate  cost-estimate 
uncertainty  (Air  Force  Cost  Analysis  Agency  [AFCAA],  2007). 

In  order  to  meet  the  requirements  of  the  Weapons  System  Acquisition  Reform 
Act  (WSARA)  of  2009,  the  SBM  would  need  to  be  able  to  produce  confidence 
interval  data  that  could  be  compared  in  the  same  manner  as  the  s-curve  analysis. 
Additionally,  the  new  method  would  need  to  be  coupled  with  historical  data  from 
previous  defense  acquisition  programs.  These  requirements  led  to  the  development 
of  the  statistical  SBM,  which  uses  simple  algebraic  formulas  and  basic  statistical 
processes  to  generate  cost-risk  curves.  Flynn  and  Garvey  (201 1 )  conducted 
research  on  historical  cost  data  to  produce  historically  driven  CVs,  lending  historical 
credibility  to  the  statistical  SBM.  This  historical  data  led  to  the  eSBM. 


B.  WEAPONS  SYSTEM  ACQUISITION  REFORM  ACT  OF  2009 

In  2009,  Congress  passed  the  WSARA  into  law  in  an  effort  to  provide  better 
oversight  and  direction  in  the  acquisition  of  defense  programs.  This  law  is  intended 
“to  improve  the  organization  and  procedures  of  the  Department  of  Defense  for  the 
acquisition  of  major  weapon  systems”  (Weapons  System  Acquisition  Reform  Act 
[WSARA],  2009,  Introduction).  With  respect  to  cost  estimation,  the  new  law  imposed 
two  important  requirements: 


1 .  The  director  of  the  Cost  Assessment  and  Program  Evaluation  (CAPE) 
shall  “issue  guidance  relating  to  the  proper  selection  of  confidence 
levels  in  cost  estimates  generally,  and  specifically,  for  the  proper 
selection  of  confidence  levels  in  cost  estimates  for  major  defense 
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acquisition  programs  and  major  automated  information  system 
programs”  (WSARA,  2009,  §  101,  subsection  2334(a)). 

2.  The  director  shall  also  “disclose  in  accordance  with  paragraph  (2)  the 
confidence  level  used  in  establishing  a  cost  estimate  for  a  major 
defense  acquisition  program  or  major  automated  information  system 
program,  the  rationale  for  selecting  such  confidence  level,  and,  if  such 
confidence  level  is  less  than  80  percent,  the  justification  for  selecting  a 
confidence  level  of  less  than  80  percent”  (WSARA,  2009,  §  101, 
subsection  2334(d)). 

These  two  requirements  drive  the  need  to  alter  the  SBM  of  cost  estimation 
into  a  more  statistically  driven  method.  In  Sections  C  and  D  of  this  chapter,  I  discuss 
the  basic  form  of  the  SBM  and  the  enhancements  needed  to  comply  with  the 
WSARA. 

C.  SBM 

1.  Non-Statistical  SBM 

The  non-statistical  SBM  method  analyzes  the  risks  of  various  aspects  of  a 
project  in  a  realistic  manner  in  the  opinion  of  the  program  manager  and  the  experts 
conducting  the  analysis.  This  method  brings  to  light  all  possible  and  reasonable 
risks  for  a  program  and  discards  the  extreme  outlier  scenarios.  Figure  1  illustrates 
the  non-statistical  SBM,  and  in  subsequent  paragraphs,  I  fully  explain  each  step  in 
the  method. 


Non-statistical  SBM 


PS 


Figure  1.  A  Non-Statistical  SBM 

(Flynn  &  Garvey,  201 1 ,  p.  7) 


The  first  step  in  this  method  is  to  add  the  estimates  of  all  the  elements  from 
the  work  breakdown  structure  (WBS)  as  the  program  is  defined  in  the  program 
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manager’s  cost  analysis  requirements  description  (CARD).  This  sum  of  all  the 
elements  of  the  WBS  is  not  adjusted  for  risk  or  uncertainty.  The  cost  estimated  from 
this  step  is  called  the  program’s  point  estimate  (PE). 

The  second  step  is  to  define  the  program’s  protect  scenario  (PS).  In  this 
step,  managers  and  analysts  define  the  sources  of  major  known  areas  of  risk  or 
uncertainty  that  have  the  potential  to  impact  the  cost  estimate.  This  PS  should  not 
include  worst-case  or  unrealistic  scenarios,  but  should  include  the  areas  in  which 
managers  should  focus  their  attention.  Once  the  PS  is  accepted,  the  cost  associated 
with  the  scenario  is  estimated  to  determine  the  program’s  PS  cost. 

The  difference  between  the  PE  and  the  PS  costs  is  defined  as  the  program’s 
cost  reserve  (CR),  as  illustrated  in  Equation  1.  By  increasing  the  program’s  PS  cost, 
the  CR  also  increases.  Conversely,  if  the  PE  increases,  the  CR  will  be  reduced  by 
the  same  amount. 


(1) 


CR  =  PS  Cost  —  PE 


Analysts  are  then  able  to  conduct  a  sensitivity  analysis  of  the  PS  cost  and  CR 
in  order  to  improve  the  model.  Once  the  model  has  been  completed,  the  program 
office  and  cost  estimators  can  finalize  all  estimates  and  generate  all  of  the  required 
reports. 

2.  Statistical  SBM 

The  statistical  SBM  expands  the  non-statistical  method  by  using  the  original 
PE  and  the  addition  of  two  inputs.  These  inputs  are  the  probability  that  PE  will  not 
be  exceeded  ( aPE )  and  the  coefficient  of  variation  of  program  cost.  By  using  the 
outputs  from  the  non-statistical  SBM,  basic  algebra,  and  simple  statistical  analysis, 
the  basic  SBM  can  be  transformed  to  produce  a  cumulative  distribution  curve  that  is 
similar  to  the  traditional  s-curve  methods. 

The  analysts  provide  the  probability  that  the  PE  cost  will  not  be  exceeded 
(aPE)  and  the  coefficient  of  variation  (CV)  of  program  cost.  The  probability  that  the 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-7- 


PE  will  not  be  achieved  is  based  on  the  historical  data  and  judgment  of  the  PM  and 
the  cost  estimators.  The  CV  is  based  solely  on  the  historical  data  from  previous 
programs.  Figure  2  shows  the  steps  of  the  statistical  SMB  process. 


Inputs  for  statistical  SBM  include  the  following: 


a.  Point  estimate  (xPE) 

b.  Probability  PE  will  not  be  exceeded  ( aPE ) 

c.  Coefficient  of  variation  (CV) 


Statistical  SBM 


These  steps  are  the  same  as  the  non-statistical  SBM  process 


Historical  Data 
Guidelines 


These  steps  are  specific  to  the  statistical  SBM  process 
Inputs  aPE  and  the  coefficient  of  variation  (CV)  are  specific  to  the  statistical  SBM  process 


Figure  2.  Statistical  SBM 

(Flynn  &  Garvey,  2011,  p.  9) 


The  CV  of  a  probability  distribution  is  the  ratio  of  the  distribution’s  standard 
deviation  to  its  mean.  Equation  2  illustrates  the  calculation  for  determining  a  CV. 

The  CV  lets  cost  estimators  and  analysts  produce  the  probability  distribution  function 
(PDF)  and  cumulative  distribution  function  (CDF). 


CV  =  D  =  -  (2) 

[i 

Two  variations  to  the  statistical  SBM  account  for  the  difference  between 
normal  and  lognormal  probability  distributions  (Flynn  &  Garvey,  2011,  pp.  10-11). 
Equations  3-6  illustrate  how  the  statistical  SBM  inputs  are  used  to  calculate  the 
program’s  mean  cost  and  cost  standard  deviation. 
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Normal  Probability  Equations 


_  _  d(xpe) 

f^COStpgm  XpE  ZpEl+D(zPE) 


^ CO  Stream 


D(xpe) 


pgm  1  +D(zpe) 


(3) 

(4) 


Lognormal  Probability  Equations 


■pgm  =  lnXPE  ~  ZPEy/\n  (1  +  D2) 

(5) 

OlnCostvgm  ~  (1  ^2) 

(6) 

The  lognormal  mean  and  standard  deviation  require  conversion  from  log- 
dollars  to  dollars  by  Equations  7  and  8  (Garvey,  2005). 


fiCost  =  e^lnC°stPgm+2<7lnCostpgrn 


(7) 


<J Cost  =  i g2plnEost:Pgm+C*lnCostpgm  ^gfflnCostp gm  


(8) 


Analysts  are  now  able  to  conduct  a  sensitivity  analysis  of  the  model  by 
altering  the  aPE  and  the  CV.  Alterations  can  create  steeper  or  shallower  slopes  to 
the  CDF,  as  well  as  shift  the  mean  of  the  function.  These  alterations  to  the  SBM 
allow  cost  estimators  to  produce  reports  that  are  similar  to  the  widely  accepted  s- 
curve  outputs. 


D.  NAVAL  CENTER  FOR  COST  ANALYSIS  STUDY  AND  THE 
ENHANCED  SBM 

In  Section  C,  I  demonstrated  the  methods  necessary  to  produce  confidence 
intervals  and  CDFs  for  the  SBM.  Alterations  to  the  aPE  and  the  CV  allow  cost 
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estimators  to  produce  reports  that  are  similar  to  the  widely  accepted  s-curve  outputs. 
The  issue  of  how  to  ground  the  SBM  in  historical  data  still  remains  problematic. 

In  201 1 ,  Flynn  and  Garvey  conducted  research  of  historical  costs  across  all 
realms  of  DoD  acquisitions  in  order  to  integrate  historical  data  into  the  SBM.  The 
research  focused  on  determining  the  appropriate  value  for  the  program’s  aPE  and 
CV. 


The  Flynn  and  Garvey  study  (2011)  derived  a  value  of  aPE  =  0.34  for 
programs  at  Milestone  B  based  on  a  historical  CV  equal  to  0.51 .  Similar  findings 
were  noted  in  the  2007  RAND  report  (Younossi,  2007),  which  noted  a  aPE  =  0.09 
based  on  an  implied  CV  =  0.26  at  Milestone  B.  These  two  reports  give  analysts  a 
bounded  region  of  aPE  to  work  within.  Now  that  a  historical  value  range  for  the  aPE 
has  been  derived,  the  only  input  left  to  consider  is  the  CV. 

In  the  Flynn  and  Garvey  study  (2011),  research  focused  on  determining  what 
actually  drives  the  value  of  the  CV  for  various  programs.  Their  first  conjecture  was 
that  cost  growth  factors  (CGFs)  have  been  historically  consistent.  The  researchers 
found  routine  underestimation  of  cost  within  the  DoD.  The  second  conjecture  was 
that  CVs  decreased  throughout  the  acquisition  process.  The  researchers  confirmed 
this  assumption  of  decreasing  CVs  as  programs  progressed  through  the  acquisition 
life  cycle.  The  third  conjecture  was  that  CVs  were  equivalent  in  all  acquisition 
programs,  regardless  of  platform.  Again,  the  researchers  proved  this  assumption  to 
be  true,  especially  from  early  milestones,  such  as  Milestone  B.  The  forth  conjecture 
was  that  CVs  would  be  lower  if  they  were  adjusted  to  account  for  quantity  changes 
and  actual  inflation.  Again,  the  Naval  Center  for  Cost  Analysis  (NCCA)  researchers 
supported  this  assumption  based  on  historical  data.  The  fifth  conjecture  was  that 
duration  would  have  no  effect  on  the  CV.  The  researchers  found  no  evidence  to 
support  this  assumption;  the  data  actually  suggested  quite  the  opposite  (Flynn  & 
Garvey,  2011). 
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E.  SUMMARY 


The  possibility  of  producing  historically  based  cost  estimates  with  minimal 
statistical  analysis  has  driven  the  desire  for  a  new  method  of  cost  estimation.  For 
any  new  method  to  be  useful  in  the  cost-estimating  community,  it  would,  as  a 
minimum,  have  to  meet  the  requirements  of  the  WSARA  (2009).  The  non-statistical 
SBM  produces  a  cost-estimation  method  that  is  bounded  by  a  scenario  agreed  upon 
by  the  program  management  team;  it  also  derives  the  program’s  PE.  The  program 
management  team’s  PE  is  coupled  to  a  probability  of  achieving  the  estimated  PE 
and  a  coefficient  of  variation  to  allow  for  statistically  adjusted  cost  estimates.  During 
Flynn  and  Garvey’s  (2011)  study  of  program  cost  estimates,  the  eSBM  gained  the 
historical  perspective  necessary  to  achieve  credibility  with  other  cost  estimators. 

The  evolution  of  the  SBM  into  the  eSBM  has  revealed  an  all-new  possibility  of  risk- 
adjusted  cost  estimation. 
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METHODS  OF  ANALYSIS 


A.  NCCA  HISTORICAL  DATABASE 

1 .  How  the  Database  Was  Created 

The  cornerstone  of  any  research  project  is  the  collectable  data.  The  NCCA 
collected  and  organized  raw  data  from  100  Selected  Acquisitions  Reports  (SARs). 
The  SARs  were  mostly  historical  DoN  major  acquisition  programs  that  were  at 
various  stages  of  completion  (Flynn  &  Garvey,  201 1 ).  The  database  organized  cost 
data  in  the  following  categories:  current-year  and  then-year  values,  quantity 
adjustments,  information  on  platform  type,  milestone  cost  estimates,  and  annual 
program  cost  updates.  By  creating  a  well-functioning  database,  the  NCCA 
established  the  groundwork  for  further  analysis. 

2.  Assumptions  Used  in  the  Database 

From  a  managerial  standpoint,  the  SAR  is  the  best  source  of  data  for  a 
program.  The  two  key  cost  estimates  present  in  the  SAR  are  the  Baseline  Estimate 
(BE)  and  the  Current  Estimate  (CE).  Although  the  cost  estimates  in  the  SAR  are  not 
perfect,  numerous  cost  studies  have  indicated  that  the  SAR  is  the  best  source  for 
comprehensive  program  data  (Flynn  &  Garvey,  201 1 ,  p.  21 ).  For  the  NCCA 
research,  as  well  as  for  follow-on  research,  the  BE  is  used  as  the  initial  cost 
estimate,  and  the  CE  of  the  last  SAR  for  a  program  is  used  as  the  final  actual  cost. 
As  stated  by  Flynn  and  Garvey  (2011)  in  the  eSBM, 

based  on  an  analysis  of  10  programs  in  our  database  dating  from  the  1990s, 
there  is  little  difference  between  the  SAR  BE,  the  program  office  estimate, 
and  the  Independent  Cost  Estimate  (ICE)  conducted  either  by  the  Naval 
Center  for  Cost  Analysis  (NCCA)  or  the  Office  of  the  Secretary  of  Defense 
(OSD),  (p.  21) 

B.  NCCA  ANALYSIS  METHOD 

In  order  to  determine  the  CGF,  the  NCCA  study  used  the  ratio  of  BE  to  CE. 

As  stated  in  Equation  2  (Chapter  II),  the  CV  is  the  ratio  between  the  sample 
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standard  deviation  and  the  sample  mean.  For  the  different  hypotheses  in  the  NCCA 
study,  the  mean  and  standard  deviation  for  each  set  of  CGFs  was  calculated  to 
determine  the  applicable  CV. 

1.  Acquisition  Milestone  Adjustment 

In  the  NCCA  study,  researchers  calculated  the  CGF  at  Milestones  A,  B,  and 
C  for  all  programs  in  the  database,  regardless  of  platform  or  type.  They  then  plotted 
the  results,  which  revealed  a  lognormal  distribution  of  CGFs  for  Milestones  B  and  C. 
There  was  insufficient  data  to  determine  the  distribution  for  data  at  Milestone  A.  The 
researchers  used  a  statistical  analogy  to  determine  the  approximate  CVs  for 
Milestone  A.  After  calculating  the  CGFs  for  each  program,  researchers  then 
calculated  the  CV  for  each  program  using  both  quantity  and  non-quantity 
adjustments  for  base-  and  then-year  dollars.  They  then  plotted  the  mean  and  range 
for  CVs  at  Milestones  A,  B,  and  C  to  reveal  lower  CVs  for  the  later  milestones.  The 
more  accurate  cost  estimation  can  be  explained  by  technology  maturation  as  well  as 
by  clearer  information  about  program  parameters.  It  is  noted  in  the  NCCA  study  that 
the  range  of  the  CVs  narrowed  at  the  later  milestones.  Figure  3  illustrates  the 
results  of  this  analysis. 


Figure  3.  Benchmark  CVs  by  Acquisition  Milestone — Quantity  Adjusted  in 

Then-Year  Dollars 

(Flynn  &  Garvey,  2011,  p.  28) 
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2. 


Platform  Homogeneity 


In  the  NCCA  study,  researchers  also  analyzed  the  difference  in  cost  growth 
between  platforms.  For  this  part  of  the  study,  researchers  separated  the  CGFs  by 
program  type.  The  type  categories  included  ships  and  subs,  aircraft,  missiles, 
electronics,  and  other.  Researchers  calculated  the  mean  and  variance  for  each 
category  to  allow  for  an  analysis  across  categories.  The  surprising  finding  of  this 
portion  of  the  study  was  that  the  CGFs  were  consistent  across  platforms  and 
categories.  At  both  Milestones  B  and  C,  the  CGFs  were  within  the  5%  alpha 
confidence  level.  This  finding  proved  that  there  is  significant  homogeneity  between 
programs.  Figures  4  and  5  illustrate  the  relationship  of  CGF  to  platform  type  at 
Milestones  B  and  C. 


Figure  4.  Milestone  B:  CGF  Means  and  Variances  (Quantity  Adjusted 

in  Then-Year  Dollars) 

(Flynn  &  Garvey,  2011,  p.  24) 
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Figure  5.  Milestone  C:  CGF  Means  and  Variances  (Quantity  Adjusted 

in  Then-Year  Dollars) 

(Flynn  &  Garvey,  201 1 ,  p.  27) 

3.  Quantity  Normalization 

To  account  for  changes  in  the  quantity  of  items  being  procured,  researchers 
in  the  NCCA  study  analyzed  the  CV  first  without  adjustment  for  changes  in  quantity 
and  then  made  adjustments  for  changes  in  quantity.  It  was  no  surprise  that  the 
adjusted  CVs  were  lower  than  the  unadjusted  CVs.  It  is  the  opinion  of  the  NCCA 
researchers  that  “changes  in  acquisition  quantity  from  a  program  baseline  are 
generally  regarded  as  beyond  the  purview  of  the  cost  analyst”  (Flynn  &  Garvey, 

201 1 ,  p.  22).  The  researchers  conducted  the  quantity  adjustment  in  two  manners  to 
determine  whether  the  adjustment  method  had  any  effect  on  the  calculated  CVs. 
First,  researchers  scaled  the  BE  to  account  for  the  actual  number  produced  or 
acquired;  next,  they  normalized  the  CE  to  reflect  the  actual  cost  if  the  original 
quantity  was  delivered.  The  differences  between  the  two  methods  proved  to  be 
insignificant;  however,  the  differences  between  the  quantity-adjusted  and  the  non- 
quantity-adjusted  CVs  were  significant. 


4.  Inflation  Normalization 


Although  SARs  are  adjusted  to  inflation,  the  values  used  are  predicted 
inflation  rates,  not  the  actual  rates  observed  during  the  time  period  covered  by  the 
program’s  life  cycle.  Inflation  rates  are  promulgated  by  the  OSD  to  ensure  that  there 
is  a  standard  set  of  inflation  values  used  across  all  areas  of  the  DoD.  The  problem 
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arises  when  the  actual  rates  are  drastically  different  than  the  predicted  rates  from 
OSD.  An  example  of  drastic  differences  between  the  two  rates  occurred  during  the 
Carter  administration  when  the  observed  rate  was  nearly  10.7%,  whereas  the  OSD 
was  forecasting  rates  between  3%  and  4%  (Flynn  &  Garvey,  201 1 ).  By  the  time  the 
OSD  began  to  change  the  rates,  inflation  was  on  the  decline  and  the  OSD  rates 
overestimated  inflation  during  the  Reagan  administration  (Flynn  &  Garvey,  2011). 
The  researchers  of  the  NCCA  study  calculated  CVs  using  the  original  inflation  rates 
and  also  using  the  actual  historical  inflation  rates.  Once  again,  they  confirmed  the 
expected  effect  of  lower  CVs  for  the  historical  inflation-adjusted  estimates. 

C.  METHOD  FOR  ACQUISITION  STRATEGY  CV  ANALYSIS 

For  my  analysis  of  the  effect  of  acquisition  strategy  on  the  CGF  and  CV,  I 
placed  each  SAR  in  a  category  corresponding  to  the  type  of  acquisition  strategy 
used  to  procure  the  item.  Once  the  programs  were  separated  into  their  respective 
acquisition  strategies,  I  computed  each  category’s  standard  deviation  and  mean. 
From  the  mean  and  standard  deviation,  I  was  able  to  calculate  the  CV  for  each 
acquisition  strategy.  To  ensure  that  the  changes  in  the  CGF  and  CV  were  not  due  to 
factors  other  than  the  acquisition  strategy,  I  adjusted  the  values  for  the  BE  and  CE  to 
account  for  changes  in  quantity  and  for  historical  inflation  rates. 

D.  METHOD  FOR  PROGRAM-LENGTH  CV  ANALYSIS 

To  analyze  the  effect  of  program  length  on  the  CV,  I  measured  each 
program’s  total  length  and  added  the  data  to  the  NCCA  database.  I  calculated  the 
CGF  for  each  program  and  then  plotted  the  CGF  against  the  program  life  span. 

Once  the  CGFs  were  plotted  with  respect  to  time,  I  determined  the  mean  and 
standard  deviation  for  the  entire  sample  set  so  that  I  could  derive  an  overall  CV  for 
program  length. 

I  also  segregated  the  CGF  into  three  general  time  frames.  The  time  frames 
were  short  acquisition  lengths  (<  5  years),  average  acquisition  lengths  (sample 
average  +/-  one  standard  deviation),  and  long  acquisition  lengths  (>  one  standard 
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deviation  longer  than  the  average).  For  each  category,  I  then  had  a  CV  calculated  to 
determine  a  relationship  between  the  program  length  and  the  expected  CV.  I 
adjusted  the  values  for  the  BE  and  CE  to  account  for  changes  in  quantity  and  for 
historical  inflation  rates  to  ensure  that  the  effects  of  the  variation  were  due  to 
program  length  and  not  inflation  or  quantity  changes. 

E.  SUMMARY 

The  NCCA  database  of  over  100  SARs,  spanning  the  last  four  decades,  is  the 
single  best  resource  of  data  on  historical  cost.  The  key  assumption  used  in  the 
NCCA  study  was  that  the  SAR  BE  would  be  used  as  the  program’s  initial  cost 
estimate,  and  that  the  CE  would  be  used  as  the  program’s  final  cost.  To  normalize 
the  initial  estimates,  the  BE  would  have  to  be  adjusted  to  account  for  changes  in 
quantity  and  changes  in  actual  inflation  from  the  estimated  inflation  rates  used  in  the 
cost-estimation  process.  A  ratio  of  the  BE  and  the  CE  were  used  to  calculate  the 
program’s  CGF.  The  NCCA  researchers  calculated  the  CV  for  various  combinations 
of  CGFs  to  determine  the  appropriate  CV  to  use  in  the  eSBM.  Additional  research 
still  needs  to  be  conducted  regarding  acquisition  strategy  and  overall  program 
length.  Comparing  the  CGFs  for  the  different  acquisition  strategies  and  program 
lengths  will  provide  further  insight  into  the  drivers  for  program  CVs. 
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IV. 


ANALYSIS 


A.  INTRODUCTION 

In  this  section,  I  describe  the  findings  for  my  analysis  of  program  length  and 
acquisition  strategy.  Section  B,  in  which  I  discuss  program  length,  is  divided  into 
three  parts:  all  programs,  completed  programs,  and  ongoing  programs.  Section  C, 
in  which  I  discuss  acquisition  strategy,  illustrates  the  problems  with  correlating  any 
one  acquisition  strategy  with  a  CGF. 

B.  PROGRAM  LENGTH 

I  conducted  the  data  analysis  for  program  length  to  determine  a  correlation 
between  program  length  and  CGFs.  I  grouped  the  programs  into  two  different 
categories:  completed  and  ongoing.  Additionally,  I  developed  a  third  group  of 
programs  to  include  all  programs.  In  the  following  sections,  I  discuss  the  findings  for 
the  claim  that  the  longer  the  program’s  length,  the  more  cost  growth  can  be 
expected. 

1.  All  Programs 

I  used  the  all  programs  group  as  the  control  to  determine  any  difference 
between  the  findings  of  Flynn  and  Garvey  (2011)  and  my  follow-on  research.  The 
sample  data  that  I  used  included  50  Milestone  B  cost  estimates.  To  verify  Flynn  and 
Garvey’s  results,  I  re-did  the  analysis  for  CGFs  and  CVs  for  all  programs.  My  results 
were  the  exact  same  as  those  illustrated  in  Flynn  and  Garvey’s  201 1  eSBM  paper. 
Their  findings  made  a  clear  connection  between  the  CGF  and  the  quantity 
adjustment.  Table  1  illustrates  the  Milestone  B  CGFs  and  CVs  for  all  programs. 
Additionally,  the  relationship  between  base-year  and  then-year  adjustments 
illustrates  the  relationship  between  the  CGF  and  inflation  for  the  same  programs. 
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Table  1.  Cost  Growth  Factors  and  CVs  at  Milestone  B  (All  Programs) 

(Flynn  &  Garvey,  201 1 ,  p.  22) 


Cost  Growth  Factors  &  CVs  for  All  DON  MDAPs  at  MS  B;  n  =  50 


(Without  Qty  Adjustment) 

(Quantity  Adjusted) 

Statistics 

Base-Year$ 

Then-Year$ 

E3ase-Year$ 

Then-Year$ 

Mean 

1.49 

1.86 

1.24 

1.36 

Standard 

Deviation 

0.94 

1.61 

0.44 

0.69 

cv 

0.63 

0.87 

0.36 

0.51 

I  analyzed  the  relationship  between  the  CGF  and  program  length  for  all 
programs.  I  assumed  that  no  further  schedule  or  performance  growth  would  occur 
for  the  programs  that  were  still  ongoing.  I  conducted  a  regression  analysis  on  all  50 
programs.  The  most  effective  regression  was  the  linear  regression  for  all  four  cases: 
non-quantity-adjusted  base-year  dollars  and  then-year  dollars,  and  quantity-adjusted 
base-year  dollars  and  then-year  dollars.  Figures  6  and  7  illustrate  the  linear 
regression  for  all  four  cases.  Figure  6  shows  a  slight  correlation  between  CGFs  and 
program  length  for  the  non-quantity-adjusted  cases.  The  correlation  is  not  as 
evident  for  the  quantity-adjusted  cases,  as  is  shown  in  Figure  7. 


The  regression  results  for  all  four  cases  yielded  insignificant  results  in 
determining  a  correlation  between  CGFs  and  program  length.  The  results  were 
insignificant  at  the  90%  level,  leading  me  to  reject  the  hypothesis  that  there  is 
correlation  between  the  two  variables.  That  is,  I  concluded  that  there  is  no 
correlation  between  the  two  variables.  The  analysis  also  failed  the  F  test  and  t  test, 
confirming  the  lack  of  correlation.  The  R-squared  and  adjusted-R-squared  results 
for  the  non-quantity  regression  had  a  minimal  significance;  however,  once  quantity 
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adjustments  were  made,  there  was  no  longer  any  noticeable  significance.  These 
results  deny  the  claim  that  program  length  has  any  effect  on  program  cost  growth. 


BY$  and  TY$  (Non  Quantity  Adjusted) 
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Figure  6.  All  Programs  Non-Quantity-Adjusted  Linear  Regression 
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BY$  and  TY$  (Quantity  Adjusted) 
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Figure  7.  All  Programs  Quantity-Adjusted  Linear  Regression 
2.  Completed  Programs 

For  the  completed  programs,  the  only  assumption  I  made  was  that  there  were 
no  outstanding  costs.  This  is  a  fair  assumption  considering  the  amount  of  time  that 
had  elapsed  for  a  majority  of  the  completed  programs.  My  first  task  was  to  conduct 
a  regression  analysis  of  the  CGFs  for  all  completed  programs,  using  the  Milestone  B 
estimates  against  the  program  length.  I  set  up  the  analysis  to  conduct  all  the  normal 
regressions  to  include  linear,  exponential,  and  logarithmic  regressions  to  produce 
the  most  significant  regression  result.  The  linear  regression  produced  the  most 
significant  results  for  both  quantity-adjusted  and  non-quantity-adjusted  values  of 
base-year  dollars  and  then-year  dollars.  Figures  8  and  9  illustrate  the  linear 
regressions  between  the  program  length,  on  the  x-axis,  and  the  CGF,  on  the  y-axis. 
Figure  8  suggests  only  minor  significance  to  the  claim  that  program  length  has  some 
effect  on  the  CGF;  however,  once  the  program  was  adjusted  for  quantity  changes, 
the  value  for  R-squared  was  reduced  to  an  insignificant  value,  as  illustrated  in  Figure 
9.  The  adjusted  R-squared  values  revealed  an  even  lower  significance  for  the 
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relation  between  the  CGF  and  the  program  length  to  claim  that  program  length  has 
an  effect  on  CGFs  for  completed  programs.  The  non-quantity-adjusted  values 
produced  results  with  modest  correlation,  but  that  were  still  not  above  the  90% 
confidence  level.  Once  I  ran  the  correlations  for  quantity-adjusted  CGFs,  the 
correlation  became  even  worse:  all  four  analyses  failed  both  the  F  test  and  t  test  for 
significance.  I  rejected  the  claim  that  program  length  affects  cost  growth  for 
completed  programs  at  the  Milestone  B  cost  estimate. 


BY$  and  TY$  (Non  Quantity  Adjusted) 


♦  BY 
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(BY) 
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Figure  8.  Completed  Programs  Non-Quantity-Adjusted  Linear  Regression 
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Figure  9.  Completed  Programs  Quantity-Adjusted  Linear  Regression 

The  next  analysis  I  conducted  was  to  determine  the  average  CGF  and  CV  for 
the  completed  programs.  Table  2  illustrates  the  CGFs  and  CVs  for  the  completed 
programs  from  the  Milestone  B  estimates.  These  values  are  consistent  with  Flynn 
and  Garvey’s  (201 1 )  findings,  as  illustrated  in  Table  1 .  My  findings  validated  the 
claim  that  changes  in  program  quantity  play  the  most  significant  role  in  determining 
cost  growth.  The  results  of  the  completed  programs  produced  more  significant 
results  than  the  results  of  all  programs  due  to  the  removal  of  the  ongoing  programs. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-24- 


Table  2.  Cost  Growth  Factors  and  CVs  at  Milestone  B  (All  Programs) 


Cost  Growth  Factors  &  CVs  for  Completed  DON  MDAPs  at  MS  B;  n  =  35 


(Without  Qty  Adjustment) 

(Quantity  Adjusted) 

Statistics 

Base-Year$ 

Then-Year$ 

Base-Year$ 

Then-Year$ 

Mean 

1.61 

2.10 

1.26 

1.40 

Standard 

Deviation 

1.07 

1.84 

0.48 

0.78 

CV 

0.66 

0.87 

0.38 

0.56 

3.  Ongoing  Programs 


The  results  for  ongoing  programs  revealed  a  negative  relationship  between 
CGFs  and  program  length.  The  negative  cost  growth  for  ongoing  programs  was 
unexpected  and  counterintuitive  to  the  results  of  the  completed  programs.  Programs 
with  the  most  time  remaining  until  completion  have  lower  CGFs.  Also,  the 
differences  between  non-quantity-adjusted  and  quantity-adjusted  CGFs  are  not  as 
obvious  because  many  of  the  programs  have  yet  to  be  affected  by  quantity 
drawdowns.  Additionally,  many  of  the  programs  have  not  been  in  existence  long 
enough  for  the  effects  of  inflation  to  become  relevant.  Figures  10  and  1 1  show  the 
negative  relationship  between  CGFs  and  program  length  for  ongoing  programs.  The 
regressions  for  the  ongoing  programs  also  showed  large  standard  errors  and  low 
adjusted  R-squared  values.  There  was  some  significance  at  the  90%  confidence 
level  for  all  four  cases;  however,  there  were  only  10  observations  that  reduced  the 
reliability  of  these  findings.  Once  again,  the  findings  of  this  research  contradict  the 
claim  that  program  length  has  any  effect  on  cost  growth. 
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BY$  and  TY$  (Non  Quantity  Adjusted) 


♦  BY$ 
□  TY$ 


- Linear  (BY$) 

- Linear  (TY$) 


Figure  10.  Ongoing  Programs  Non-Quantity-Adjusted  Linear  Regression 


Figure  11.  Ongoing  Programs  Quantity-Adjusted  Linear  Regression 
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Table  3  shows  the  average  CGFs  and  CVs  for  the  ongoing  programs.  Even 
for  the  ongoing  programs,  the  CGFs  and  CVs  for  the  quantity-adjusted  values  were 
within  the  range  of  the  completed  program  threshold. 


Table  3.  Cost  Growth  Factors  and  CVs  at  Milestone  B 

(Ongoing  Programs) 


Cost  Growth  Factors  &  CVs  for  All  DON  MDAPs  at  MS  B;  n  =  10 


(Without  Qty  Adjustment) 

(Quantity  Adjusted) 

Statistics 

Base-Year$ 

Then-Year$ 

E3ase-Year$ 

Then-Year$ 

Mean 

1.25 

1.33 

1.27 

1.35 

Standard 

Deviation 

0.41 

0.46 

0.37 

0.46 

cv 

0.32 

0.35 

0.29 

0.34 

C.  ACQUISITION  STRATEGY 

1.  Types  of  Acquisition  Strategies 

The  analysis  of  acquisition  strategy  resulted  in  more  questions  and  no 
definitive  answers.  The  problem  with  relating  any  one  acquisition  strategy  to  a  CGF 
is  the  complexity  of  the  acquisition  system  and  modern  program  management. 
Program  managers  use  a  mixture  of  contract  types  during  the  Milestone  B  and 
beyond  the  contracts  negotiation  stage.  For  the  current  ongoing  programs,  each 
program  uses  a  mixture  of  firm-fixed-price  (FFP),  fixed-price-incentive-fee  (FPIF), 
cost-plus-fixed-fee  (CPFF),  cost-plus-incentive-fee  (CPIF),  and  cost-plus-award-fee 
(CPAF)  contract  types.  The  problem  of  assigning  an  acquisition  strategy  to  a  single 
contract  type  becomes  more  difficult  when  the  issue  of  awards  and  incentive  fees  is 
taken  into  account.  Awards  and  incentives  are  written  into  contracts  at  the 
beginning  but  not  awarded  until  contract  completion. 
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2. 


Continuation  of  Research 


The  next  step  in  trying  to  determine  how  acquisition  strategy  affects  cost 
growth  is  to  break  down  cost  growth  for  each  contract  type  into  individual  CGFs. 
Once  the  individual  CGFs  have  been  determined,  they  can  be  weighted  to  account 
for  the  relative  amount  each  contract  contributes  to  the  total  program  cost.  Breaking 
down  the  programs  to  the  next  lowest  level  allows  for  further  insight  into  the  drivers 
of  cost  growth. 

D.  SUMMARY 

To  the  95th  percentile  confidence  level,  I  have  rejected  the  hypothesis  that  as 
program  length  increases  so  does  the  CGF.  Furthermore,  the  findings  of  this 
research  have  confirmed  the  findings  of  the  NCCA  study  on  CGFs  and  historical 
CVs.  The  analysis  of  program  length  yielded  no  significant  relationship  between  the 
program  length  and  CGFs.  The  analysis  of  program  length  reiterated  the  effects  of 
quantity  changes  on  CGFs  as  a  significant  driver  for  cost  growth.  Additionally,  the 
effects  of  inflation  on  program  cost  growth  continue  to  plague  the  cost-estimation 
community.  Neither  the  changes  in  quantity  nor  the  changes  in  inflation  rates  are 
under  the  control  of  the  cost  estimators;  however,  these  changes  must  be  accounted 
for  in  the  cost-risk  profile. 
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V. 


CONCLUSION 


A.  SUMMARY 

In  Chapter  I,  I  discussed  the  basis  for  this  research  and  the  issue  of  risk- 
adjusted  cost  estimation  using  the  Monte  Carlo  simulation  method.  The  defining 
issues  with  cost  estimation  at  Milestone  B  include  the  uncertainty  of  a  clear  program 
direction,  the  difficult  statistical  processes,  and  the  unbounded  reality  of  possible 
consequences.  These  difficulties  led  to  the  need  for  a  new  method  of  calculating 
risk-adjusted  cost  estimation.  A  new  method  would  be  beneficial  to  everyone 
involved  in  the  acquisition  process,  from  the  MDA  to  the  cost  estimators. 

In  Chapter  II,  Literature  Review,  I  described  Paul  Garvey  and  Brian  Flynn’s 
(201 1)  findings  as  a  more  effective  method  of  risk-adjusted  cost  estimation.  This 
chapter  started  with  the  necessary  requirements  defined  in  the  WSARA  (2009).  This 
act  tasks  the  director  of  CAPE  with  issuing  guidance  on  the  appropriate  confidence 
intervals  to  be  used  and  with  justifying  their  use  if  that  level  is  below  the  80% 
threshold.  Garvey’s  (2005)  SBM  provided  the  framework  for  a  simplified  method  of 
risk-adjusted  cost  estimation  using  the  statistical  SBM.  Flynn  and  Garvey’s  (2011) 
study  on  CGFs  and  CVs  led  to  enhancements  in  the  SBM  that  aligned  the  new 
eSBM  with  the  requirements  of  the  WSARA  (2009). 

Chapter  III,  Methods  of  Analysis,  included  two  parts:  what  has  been  done  in 
the  past  and  what  this  thesis  does  differently.  The  NCCA  study  brought  to  light  the 
issues  of  inaccurate  inflation  estimates  and  quantity  adjustments  as  significant 
reasons  for  cost  growth.  The  NCCA  study  also  debunked  the  theory  that  there  are 
secular  differences  in  cost  growth  across  program  types.  The  methods  used  in  this 
thesis  were  aimed  at  relating  cost  growth  to  program  length  and  acquisition  strategy. 


I  based  my  analysis  in  this  thesis  on  two  hypotheses:  first,  cost  growth 
increases  as  program  length  increases;  and,  second,  different  acquisition  strategies 
yield  different  rates  of  cost  growth.  I  rejected  the  first  hypothesis  with  a  high  degree 
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of  confidence  because  I  noted  no  correlation  between  program  length  and  CGF.  My 
analysis  of  program  length  did,  however,  further  support  Flynn  and  Garvey’s  (2011) 
findings  that  inflation  and  quantity  adjustment  are  the  most  significant  sources  of 
cost  growth.  During  my  analysis  of  the  second  hypothesis,  I  discovered  that  no 
single  acquisition  strategy  is  used  in  contracting  an  MDAP.  The  complexity  of 
acquisition  strategy  made  the  second  hypothesis  impossible  to  validate  in  the 
manner  that  I  illustrated. 

B.  RECOMMENDATIONS 

I  recommend  two  main  areas  for  continued  research  on  historical  cost  growth 
and  potential  cost  drivers. 

The  first  major  area  that  needs  to  be  researched  is  in  contract  management 
and  in  the  methods  used  as  possible  cost  drivers.  This  area  should  include  a 
breakdown  of  each  program  to  the  contract  level  to  determine  the  cost  growth  for 
different  contact  types,  and  then  the  contract  types  across  programs  could  be 
compared  as  potential  links  to  cost  growth. 

The  second  area  in  which  research  could  be  beneficial  is  the  effect  of  awards 
and  incentives  on  cost  growth.  An  analysis  of  what  percentage  of  cost  growth  is 
accounted  for  in  awards  and  incentives  could  lend  insight  into  how  to  more 
accurately  estimate  awards  and  incentives  for  MDAPs. 

Other  areas  in  which  research  is  still  needed  in  the  risk-adjusted  cost- 
estimation  arena  include 

1 .  how  the  eSBM  compares  to  the  Monte  Carlo  simulation  at  later 
milestones,  and 

2.  how  to  account  for  and  eliminate  PM  bias  during  the  cost-estimation 
process. 
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Analysis  of  LAV  Depot  Maintenance 
Army  LOG  MOD 
ASDS  Product  Support  Analysis 
Cold-chain  Logistics 

Contractors  Supporting  Military  Operations 
Diffusion/Variability  on  Vendor  Performance  Evaluation 
Evolutionary  Acquisition 
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■  Naval  Aviation  Maintenance  and  Process  Improvement  (2) 

■  Optimizing  CIWS  Lifecycle  Support  (LCS) 

■  Outsourcing  the  Pearl  Harbor  MK-48  Intermediate  Maintenance 
Activity 

■  Pallet  Management  System 

■  PBL  (4) 

■  Privatization-NOSL/NAWCI 

■  RFID  (6) 

■  Risk  Analysis  for  Performance-based  Logistics 

■  R-TOC  AEGIS  Microwave  Power  Tubes 

■  Sense-and-Respond  Logistics  Network 

■  Strategic  Sourcing 

Program  Management 

■  Building  Collaborative  Capacity 

■  Business  Process  Reengineering  (BPR)  for  LCS  Mission  Module 
Acquisition 

■  Collaborative  IT  Tools  Leveraging  Competence 

■  Contractor  vs.  Organic  Support 

■  Knowledge,  Responsibilities  and  Decision  Rights  in  MDAPs 

■  KVA  Applied  to  AEGIS  and  SSDS 

■  Managing  the  Service  Supply  Chain 

■  Measuring  Uncertainty  in  Earned  Value 

■  Organizational  Modeling  and  Simulation 

■  Public-Private  Partnership 

■  Terminating  Your  Own  Program 

■  Utilizing  Collaborative  and  Three-dimensional  Imaging  Technology 

A  complete  listing  and  electronic  copies  of  published  research  are  available  on  our 
website:  www.acguisitionresearch.net 
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